fix: merge case-duplicate github repos and enforce unique lower(url) [CM-1344] by mbani01 · Pull Request #4383 · linuxfoundation/crowd.dev

mbani01 · 2026-07-22T11:41:58Z

This pull request introduces a migration to clean up duplicate GitHub repositories in the repos table, which arose due to case sensitivity issues in URLs. The migration consolidates duplicate entries by keeping a single canonical (lowercase) row per repository, re-points associated data, and enforces case-insensitive uniqueness for GitHub URLs going forward.

Duplicate GitHub repository consolidation:

Identifies groups of duplicate repositories (differing only by URL case) and selects a keeper per group, preferring the all-lowercase variant when present.
Re-points package_repos and repo_docker to the keeper repository, ensuring only one link per package/group combination, and removes redundant links.
Deletes derived data (repo_scorecard_checks, security_contacts, repo_activity_snapshot) associated with loser repositories, as these will be regenerated as needed.
Removes the loser repository rows from the repos table and normalizes the surviving repository URL to lowercase if needed.
Adds a partial unique index to enforce case-insensitive uniqueness of GitHub repository URLs, preventing future duplicates.

…ueness Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

cursor · 2026-07-22T11:42:05Z

PR Summary

High Risk
Large-scale destructive data migration on core repos/package_repos with Tinybird CDC implications; incorrect keeper or link logic could break package–repo provenance or duplicate security contacts until sweeps recover.

Overview
Adds two Flyway migrations to clean up ~40k case-variant duplicate GitHub repos rows left after CM-1305 re-pointed package links but did not remove losers.

The first migration runs atomically: picks a keeper per LOWER(url) (prefer all-lowercase, else lowest id), dedupes package_repos using the same ranking as bestRepoLink, re-points loser links via insert+delete (not UPDATE) for Tinybird CDC, updates repo_docker and lowercases denormalized packages.repository_url, drops derived loser data, deletes loser repos, then lowercases surviving mixed-case GitHub URLs.

The second migration adds a partial unique index repos_github_lower_url_uq on LOWER(url) for host = 'github' CONCURRENTLY, split out so the merge stays transactional and workers stay writable; index build failure also verifies no duplicates remain.

^{Reviewed by Cursor Bugbot for commit ac4dc81. Bugbot is set up for automated code reviews on this repo. Configure here.}

CLAassistant · 2026-07-22T11:42:08Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Copilot

Pull request overview

Consolidates case-duplicate GitHub repositories and prevents recurrence.

Changes:

Re-points dependent records and removes duplicate repositories.
Normalizes surviving GitHub URLs.
Adds case-insensitive uniqueness enforcement.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

backend/src/osspckgs/migrations/V1784718693__merge_case_duplicate_github_repos.sql:101

url is the Tinybird repos sorting key (services/libs/tinybird/datasources/repos.datasource:74). This update therefore inserts a new lowercase key but cannot replace the previously ingested mixed-case key; FINAL keeps both, and joins by repo ID can duplicate downstream results. Include a Tinybird cleanup/rebuild or tombstone path for the old URLs.

UPDATE repos r
SET url = LOWER(r.url), updated_at = NOW()
WHERE r.host = 'github'
  AND r.url <> LOWER(r.url);

Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit eff5795. Configure here.}

cursor · 2026-07-22T12:30:59Z

+FROM repos r
+WHERE r.host = 'github'
+  AND p.repository_url = r.url
+  AND p.repository_url <> LOWER(p.repository_url);


Case-sensitive package URL join

Medium Severity

The migration lowercases packages.repository_url only when it equals a GitHub repos.url byte-for-byte. GitHub URLs are case-insensitive, so a package can refer to the same repo as a row in repos while using a casing variant that was never inserted. Those rows skip the update and keep a mixed-case repository_url after repos is normalized, breaking the intended alignment with canonical lowercase URLs.

^{Reviewed by Cursor Bugbot for commit eff5795. Configure here.}

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

backend/src/osspckgs/migrations/V1784718693__merge_case_duplicate_github_repos.sql:112

This key-mutating update is not CDC-safe for Tinybird: repos.datasource uses url as its ReplacingMergeTree sorting key (services/libs/tinybird/datasources/repos.datasource:72-75), so the lowercase event cannot replace the old mixed-case key under FINAL. The migration mentions a packageRepos rebuild, but the deployment must also rebuild repos (and account for the deleted derived rows), otherwise the duplicates remain visible in Tinybird.

UPDATE repos r
SET url = LOWER(r.url), updated_at = NOW()
WHERE r.host = 'github'
  AND r.url <> LOWER(r.url);

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+UPDATE repos r
+SET url = LOWER(r.url), updated_at = NOW()
+WHERE r.host = 'github'
+  AND r.url <> LOWER(r.url);


fix: merge case-duplicate github repos and enforce lowercase url uniq…

c918213

…ueness Signed-off-by: Mouad BANI <mouad-mb@outlook.com>

mbani01 self-assigned this Jul 22, 2026

Copilot AI review requested due to automatic review settings July 22, 2026 11:41

Copilot started reviewing on behalf of mbani01 July 22, 2026 11:42 View session